Yeah, sorry about that, Matthew is still on the way. Okay, hang on, reply coming. Yeah, it's okay. U oh, I hate these things. Voila. Okay. So Um we can maybe get a head start. Um have you been talking to Matthew about what you've been doing? Or Yeah. Yeah, yeah. Yeah. Yeah, which was the intuitive feeling. Yeah. Yeah. And is and the r result was no. But uh bu but the question is if is that result no because it's no or is it also just because there's not really enough data to be sure about anything? Sure. Yeah, okay. Yeah. Yeah. Yeah. It's not gonna yeah. Yeah. Yeah, yeah. This is so sort of a feeling that we had but hadn't shown. I think I mean i in uh from uh from an application point of view it might be interesting to make sure that your vocabulary contains all the words on the slides. But from a research perspective it's not interesting. There'd be very few. Well, this is h this is how the dictionary's calculated. It calculated such that if you don't have the word it doesn't affect to a great degree the the the word error rate. So Yeah. Sure. Yep. M like yeah. Yeah. Yeah. But th yeah. No. Well, this is yeah. That's yeah. Yeah. Oh sorry about that. Yeah. Of course. Yeah, yeah. Yep. 'Cause it's not gonna make any difference. Yep. Yep. Hmm. Yeah. Hmm. I mean the it's probably very dependent on the speaker as well. Some people have a habit of just reading what they've got on their slides and some people are com are are well, or or or quite purposefully talk about different things so that they've sort of got multi-modality of you know. But so I mean I guess that's not a surprising result then. The um Yeah. Yeah. S Yeah. Yeah. Yeah, sure. We s go straight to the semantics of the yep. Hmm. Yes. Maybe. Okay. Oh okay. Well I mean uh it just comes down to the to the mere fact that word error rate is just take the words. How many of them did you get right. Take plus or minus two or three or five or ten words doesn't really make any difference. So yeah. Even if those words at the end of the day would be quite important in any sort of search of the transcript. But at the end of the day why wouldn't you just use the slides to search the transcript? I if you have th slides, then yeah. No. Y you'd ju I mean the uh I mean it so I mean have you been doing other things then? Have you have you been thinking about what you would like to to do in in in place of this? Or or or leading on from this, given what you've learnt? 'Cause ours is strict. Yeah. Yeah. Associated? Yeah. Sure. The focus. Yeah, okay. Yep. Okay. M meeting action sort of focus. Yep. Hmm. Yeah. Yep. Yeah. Yeah, sure. Yeah. W Well Yeah. Ye yeah, okay. So I mean uh you could begin with s saying the a um the simple task of just determining whether or not the speech is related to the slide content, and which is sort of building upon what, you know, Dong and others have worked on in terms of, you know, is it discussion, monologue, from the original M_ four data collection. Uh a y are are you also considering that you could actually look at the relationship between meetings for instance? Um well okay, well I mean it if if if these similar sort of phrases or words were discussed in this meeting and were also discussed in the previous meeting, then you sort of have a a link between meetings for instance. I don't know I don't know how this you have to ches the check the statistics. But I I mean 'cause obviously you've gotta think towards what, you know, this this won't occupy ab ab you know, won't occupy you for three years or whatever just on this I guess. But have to look at all the future and so just a s Yeah, yeah. Okay. Yeah. Yeah, okay. Yeah. and yep. Yep. Yeah. Well y yeah. Sure. Yeah. Yeah. I mean uh it's it's and because we don't have all the data collected or annotated yet, it's it's very difficult to know, isn't it. Yeah. Yep, yep, yep. Hmm. Yeah, sure. Or b or as you say, back-channel versus yeah. 'Cause I th think that's quite interesting. I mean 'cause it's an awful lot of speech activity, which is yeah. Sure. Okay. Okay. So I mean as far as that you've got everything to do that, I presume. Um you don't need any extra stuff from us in the immediate future? Or I mean that's good. That's good. But I mean it it Uh yeah I wouldn't get too distracted I guess on d d if you've got two tracks, I'd like to Yeah. Um Yeah. Hmm. But I And uh the k and the question is also are there other sort of more sensible ways of doing it. For instance um gaze tracking. I mean if if people are looking at the slides Yeah, sure. And it has specific to the environment and all that sort of thing. Yeah. Yeah. Yeah. Mm. Uh but I mean for instance it should be linked to the other stuff that you've been doing on um call routing for instance. And and um, you know, uh error merit err error measures, yes. I mean I think I mean I I think that it's possible to draw some sort of relationship between the two, as we were saying. I mean maybe if you you don't measure things in terms of word error rate, maybe this sort of information does actually mean something. Given that fifty percent of the words are not actually interesting. Um so I mean W Well w uh w No, it doesn't solve the problem, but it's f what is it? Yeah. Hmm. Sure. Mm. Yeah. I think I think it's maybe an interesting test scenario and we know that you're sort of basing something around speech and and text. I think there needs to be one sort of extra higher goal above that so that you can motivate future research. Hmm Yeah. Yeah, this was Yeah. For English it's especially low. Yeah. Yeah. Speci Yeah. Yeah. Well another thing you have to remember is these models have been tuned to include AMI data. So I mean perhaps if you had slides and and presentations to do with something that was off topic, you didn't have you hadn't seen any prior data, then you might see a bigger contribution as well. But I mean essentially you've actually sort of included the information from the slides in the language model. Yeah. Okay. Okay, they're pre pre prepared. Okay. Yeah, yeah. It's not there. Okay. Y the audio quality though. Yeah. Yeah. Was good to yeah. Yeah. Ah. Yeah, they had like uh y uh y they, you know, they had this set-up with the uh the microphones. So someone every it there was one microphone shared between every two people. But they had deals with people forgetting to turn the microphone on, forgetting to turn it off, uh f it was it was meant I think they they chose that lecture theatre possibly with the idea that it would be good for collecting data. I mean I uh I'm sure they could have probably had University of Edinburgh host it or whatever. But I th I think the, you know, the H_C_I_ part of it didn't quite work out to how they planned. So ap Yeah. Um it is I mean it's same as us putting on a the headphones and whatever. I mean even that minor thing. I mean any any audio captured during that period is i is rubbish. But there's no sort of system that's really built to d to be able to say when you're you're capturing that. Okay. The Uh what will that incorporate? Will it incorporate like uh an S_M_A_ and uh Yep. Okay. Yeah. Okay. Okay. So no no uh microphone arrays or anything like that. I guess that's quite a that's a quite a bi big deal for setting up the recordings, I guess. Yeah. Yep. As you want, yep. Okay. Yeah, but let's pretend that it's not going to happen in any foreseeable future. Wh uh the I mean I'm just thinking from practical perspectives, you'd you'd have to the the AMI recognition system that we have now is either for these microphones or for the microphone array. There's no lapel system trained for instance. So I mean that in itself, it would I mean you'd have some fun training some more speech recognition models. But Yeah. Yeah. Yeah. Yeah, that's Sure. Yeah. It's what you should be using for this type of task. I think Yeah. And every time we d we generate a new corpus and try to carry out new research we discover sort of areas that are too controlled, you know. I mean the AMI ones were a lot better in term of that than M_ four or whatever. But still. I mean es especially when you get into higher level sort of uh interpretation, it's yeah. Oh well. I'm uh I'm just curious, do the like the ICSI meetings, do they have anything other than the audio? Do they have slides or anything along that I don't think they would. Okay, I'm just thinking whether or not there an are any other sources, I mean maybe CHIL data? 'Cause it's I think it's gonna be more like the TAM type situation. So Mm-hmm.. I don't know. But I mean given that it's coming from the same basis as AMI, I I expect there has to be some sort of relatively f liberal sort of distribution policy. Yeah. Yeah. The one thing about that, and this is something that came up in the last NIST evals, was the fact that it's almost too uni-modal. You've just got one lecturer talking. So once again you don't see the the complex sort of levels of interaction. I mean uh y uh there's no way of getting around that. I mean you just have to think of a a task a challenge that is relevant to that type of recording, which might mm you might use the same methods as what we've been pr talking about today. But the actual sort of what you're trying to measure might need to be slightly slightly different I guess. I don't yeah, you know, in terms of in terms of, say, measuring whether or not they're talking about the slides or off channel, I mean you might find ninety eight percent of your data is directly referring to your slides in a t in a TAM. That is uh is in TAMs. Yeah, but I'm I'm meaning if you move on to these corpora, yeah. So you might wanna be doing something else basically, yeah. Yeah. Yep. Yeah. Yep. Yeah. And uh and at the s and at the same end of the day, you're probably going to be using or learning about the same techniques that you'll need for doing other things along these lines. So Okay. It well it's it's to make a proposal basically for your yep uh This n Okay. Yep. The sp Mm-hmm. Yeah. Sure. Sometimes it's a lot more. Yeah. Yeah. Mm yeah. Yeah. Well, you'd uh sure. At l at at least spot if they forget to talk about something and it won't let them continue on to the next slide until they've gone through all the points. Yeah. Yeah. Yeah. Well uh that that was just through sheer weight of content though. W B yeah. As in five, six, you know number where where the graphemic yeah, well I yeah. Yeah. Although I remember discussing this like um 'cause uh when Neville was working on his his stuff, which was just based on the slides, if um he he asked me all you know, could I pre-segment the slides for him uh, you know, because you wanted to s you wanted to have the topic segmentation. And I said well just look at the the headings. Because anybody generally who writes the slides yeah. It's not okay. It's true for me then. Okay. Yep. Okay. Yeah. Okay. Okay. Uh it yeah, it depends, yeah. That's a stylistic thing as much as anything. Yep. Yeah. Hmm. Mm. Yeah. Which uh whether or not that's a good or a bad thing. It depends on the talk. Yeah, presentation, yeah. Yeah. Yeah. Something we're taught at university from a very young age. the structure. Yeah, yeah, sure. It's all subjective. Hmm. Okay. Okay. Okay. Yeah, yeah. It's quite strictly imposed now. Ye Yeah. After the after the hard drive crash we took the time to actually rebuild the the the system and actually train 'cause originally we sort of had the idea of oh we'll use a recognizer that's been trained on a large amount of telephone speech data and that should give us a better result. I just as a contrast, I trained one specifically on this data. Results were almost identical um which says to me there's something wrong with the data rather than and it's nothing serious. It's simply that it hasn't been um chunked it or or segmented in a way that's typical of speech resources. in terms of um you can have like uh a two second utterance, uh credit cards, and then twenty seconds of silence coming after it, which in which when you f when you first think about it, you just think oh silence, whatever, nothing will happen. But a lot of the normalization that we use Uh yeah, C_M_S_ C_ C_V_N_. Uh so w we need a decent end point detection system running before we can we can we can do that. And we're converging on having one now just in in sort of tandem with this development of this web based recognizer. Um but of course unfortunately all these things take time. Um but I I think we should see a decent sort of improvement then. Not just in terms of no longer having recognition errors whenever silence appears. 'Cause you get that immediate benefit that you don't have any insertion errors. But also it should actually affect the recognition on the speech segments as well. Th The Yeah. Yeah. Mm that's the hard p oh I might uh given the trouble we've had with this corpus I can't make any promises. That's the intention, yep. Sure, sure. We trained the system just on this data. Like there were there were six thousand utterances that weren't used for uh the call routing side of things. I think they th there were thirteen thousand in the entire corpus. Six t half of them roughly we've thrown away. So use that for acoustic training. And the results from that were almost the same as what we got for the C_T_S_ model. Y yeah. But one can imagine a commercial system uh that that that s Steven Cox used from Nuance probably had all these things, built in end point detection, et cetera, et cetera. So Mm. Yeah. Yeah. And there's also just things like tele uh receiver noise at the beginning and or at the end of the and I mean these these are things you can get rid of if it's a Mm. Yep Mm okay, this is f okay. Okay. We can just run some generic yeah. Well I don't mind running it. I've got all the scripts set up. So Well I'm not I'm not sure I see the point actually. Uh Mm. I don't know. Yeah. Whatever, I don't know thing th I don't know whether that a I don't know whether that's a We might as well. 'Cause I mean we've up until now uh you haven't been running any recognition on and neither did Jean-Yves on that on that data. Yeah. Yeah. Oh sure, but I mean the scripts are so automated, it's just a matter of sitting uh uh submitting a job to the cluster and I don't I don't see the point of that. I don't mind running it. That'd it'd be like uh half an hour's work in a day to run the recognition scripts. I mean I I I I know you're saying. Artem should familiarise himself with the A_S_R_ systems or whatever. Fine. But I don't uh I d I don't think this is a point that's worth sort of worrying about at this stage anyway. I mean Well 'cause I'm I'm sort of assuming that that we're gonna have so an end point detection regardless of this working on this Nuance corpus. So We need it for AMI. We need it for um the online demo. For uh all these things, yeah. It's it's a lot more difficult than you would think. Well, it's speech and any and any other class of sound. That's the problem. It's And is And any other class, yeah. If it was just silence it'd be easy. But uh um and in controlled places you can't just assume it's it's, you know, any energy below a certain threshold. And this is what everybody sort of claimed the problem was solved with years ago. But Well, I mean this is once again a Yeah, sure. But I mean this is once again a very constrained task again. I mean it's it's it's you you're almost assured it's only gonna be a single speaker on the microphone. So I mean the the only sorts of noise you're gonna have to deal with uh is is is hiss or impulsive I would say. Or or that would be a good start. No, no. Yeah. yeah. It was showing that. Um you haven't heard from him I take it either? No. Okay. Well it'd be nice to get the thesis I think. There's a sort of just a research report sort of basis for this. Yeah. But Okay. yeah. apply some pressure. Uh n no it should be looking towards the start of next year basically. The is it so I mean you should Okay, which is Yeah. And it's nice to get some feedback on I mean that feedback will probably come after your proposal possibly. But um Yeah. Yeah, yeah, yeah. Exactly. N yeah, sort out any bugs or whatever. Yeah. Yeah. I I I'd prefer to go back to the C_T_S_ models I think as well 'cause I think I think they did work a little bit better. So yeah. Yep. Hayes is another one isn't it? Oh okay, sorry. I was like 'cause I saw you sli I saw you looking at your watch before, and I like wow. Gonna have to s Have this AMI corpus linked in no time. Oh okay. Uh not normally, yeah. It could go to thirty seven, thirty eight. Yeah. Yeah, we've been told. Okay. Fine.